Skip to content

feat(openfga): add psk auth to config - #656

Open
NJona wants to merge 11 commits into
multi-dc-10-multi-dc-tests-docsfrom
jona/openfga-psk-auth
Open

feat(openfga): add psk auth to config#656
NJona wants to merge 11 commits into
multi-dc-10-multi-dc-tests-docsfrom
jona/openfga-psk-auth

Conversation

@NJona

@NJona NJona commented Aug 7, 2026

Copy link
Copy Markdown
Member

For multi dc we need to expose openfga from one dc to be reused by all others (one central required component). This can now be done by enabling gateway and setting the host.

Everything in this PR was already validated manually in a multi dc setup.

Related PRs that were used for the testing:

NJona and others added 10 commits August 5, 2026 15:17
Until now a bootstrapped project was implicitly a single data center:
its nodes, gateway IPs, config paths and domains all lived directly on
CodesphereEnvironment. Multi-DC support needs more than one of each, so
this introduces the DataCenter type that holds everything which must
differ per data center, while project-level state (project, VPC,
jumpbox, shared postgres node, registry) stays on the environment.

BuildDataCenters derives the layout from the flags: one entry today, and
with --multi-dc a second one that shares the first's PostgreSQL server.
The primary data center keeps an empty resource-name suffix, so every
name, path and domain a single-DC bootstrap produces is unchanged.

Nothing consumes the layout yet — the callers are migrated in the
following commits. Two mechanisms keep that migration safe:

- ensureDataCenters derives the layout on first use and adopts state a
  caller passed through the legacy top-level environment fields, so
  every entry point works whether or not Bootstrap ran first, including
  infra files written before multi-DC support.
- mirrorPrimaryDataCenter projects the primary data center back onto
  those fields before the infra file is written, so cleanup and
  restart-vms keep reading what they always have.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
The VM list was a package-level global describing one data center. It
becomes a function of the environment: the project-shared jumpbox and
postgres VMs plus each data center's three Ceph and three k0s nodes,
whose names carry that data center's suffix. A single-DC bootstrap
therefore still gets exactly ceph-1..3 and k0s-1..3.

EnsureComputeInstances routes each instance into its own data center by
the definition's DataCenterID instead of inferring placement from the
VM's tag, and sorts each data center's nodes independently — the install
config assigns roles by index.

restart-vms resolves the valid VM names from the infra file's data center
layout, so a node of a second data center can be restarted by name. It
now also reads the infra file when --project-id and --zone are given,
best-effort, since only the file knows the layout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Migrates the infrastructure steps from the single implicit data center
to the layout: root login and host configuration run over every data
center's nodes, each data center reserves its own gateway, public
gateway and SSH proxy IP under a suffixed name, and each gets its own
k0s configuration script patching its own gateway services.

InstallCodesphere and RunK0sConfigScript now loop over the data centers
in ascending order and log a step per data center. The order matters
once there is more than one: the primary data center's install creates
the database, roles and schema the others reuse, and a k0s script can
only patch gateway services an install has already created.

The install command moves into the exported InstallCommand, since the
CLI prints it for the operator when the bootstrap does not install
itself, and it now names the data center's own config, vault and age
key.

EnsureHostsConfigured also creates /etc/codesphere/secrets up front on
every node: the installer uploads a data center's age key to that fixed
path but only creates its own configured secrets.baseDir, so for a data
center whose baseDir differs the upload target would not exist.

Behaviour for a single data center is unchanged — same VM names, same IP
names, same script, same install command.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Workspaces resolve per data center, and so does the platform: the
frontend asks <dc-id>.<codesphere.domain> for the configuration of the
data center a workspace lives in. OMS only created cs.<base-domain> and
its wildcard, both pointing at the first data center's gateway, so with
more than one data center the second one's endpoint resolved to the
first's gateway, which has no route for that host — every browser
request for it was reset, and since the frontend fetches that config
before rendering, the whole UI failed.

EnsureDNSRecords now creates, per data center, its workspace hosting
names and SSH proxy name pointing at its own public gateway and SSH
proxy, plus <dc-id>.cs.<base-domain> and its wildcard pointing at its
own platform gateway. The per-data-center platform names are only
created when there is more than one data center: a single one is the
primary, which cs.<base-domain> already resolves to.

The records that were created are recorded in the infra file, so cleanup
deletes exactly those. DeleteDNSRecordSets therefore takes the record
list instead of a base domain, and cleanup falls back to deriving the
names for infra files written before this and for a cleanup driven only
by --project-id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Config and vault generation worked on b.Env.InstallConfig and the single
b.icg. Both become per data center: every step takes the DataCenter it
generates for and reads its nodes, IPs, paths and config manager from
there. The exported Ensure*/Update* entry points keep their signatures
and operate on the primary data center, so step names and existing
callers are unchanged.

Two details worth pointing out:

- secrets.baseDir is now the data center's own directory. The installer
  resolves the vault it reads and writes back from that path, so sharing
  it between data centers would let one data center's kubernetes and
  ceph steps overwrite the other's kubeconfig and Ceph credentials.
- the local config, vault and remote paths come from the data center, so
  a second data center writes config-dc2.yaml rather than overwriting
  config.yaml.

Pure refactoring: a single-data-center bootstrap produces the same
config and vault as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
The registry was written straight into the single install config, so
which registry the nodes pull from was decided per config rather than
per project. It now resolves onto the environment
(ContainerRegistryURL plus credentials) and updateInstallConfig applies
it to every data center's config and vault. All three registry types go
through the same field, which also fixes the artifact registry never
recording its URI on the create path.

EnsureLocalContainerRegistry is split in two, because it early-returned
when the registry was already running and thereby skipped distributing
the registry certificate. A re-run that adds a data center hits exactly
that path, and its nodes would then fail every pull with "certificate
signed by unknown authority". Starting the registry stays conditional;
distributing the certificate now always runs, over every data center's
cluster nodes, which is safe because it is idempotent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Wires up multi-data-center support: --multi-dc bootstraps a second data
center in the same project, sharing the VPC, jumpbox and PostgreSQL
server but running its own Kubernetes and Ceph cluster.

The secondary data center's config and vault are derived from the
primary's after its secrets exist, which is why Bootstrap handles it in
a loop after the primary's config is written:

- the config is cloned and its data-center-scoped fields cleared, so the
  installer's dataCenters topology, the domains and the shared registry
  are inherited while the ingress CA and cephadm key are regenerated;
- the vault is derived through DeriveDataCenterVault, keeping the
  postgres roles and token keys and dropping the per-cluster secrets;
- postgres becomes mode: external pointing at the shared server's
  internal IP, which is what its certificate's only SAN carries, and
  "postgres" is appended to operations.skip in the config so manual
  re-runs on the jumpbox skip it too.

A post-generation check fails the bootstrap if secrets that must match
across data centers diverged, or if a per-cluster secret was inherited
instead of regenerated — both would otherwise surface long after the
fact, against a live shared database.

Installs run strictly sequentially in ascending data center order, since
the primary's install creates the database, roles and schema the others
reuse. validateMultiDC rejects the combinations that cannot work
(--write-config=false, an explicit --datacenter-id, an empty datacenter
name), and the CLI prints one install command per data center in the
required order.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Covers a full --multi-dc bootstrap in one spec: two data centers with
their own nodes, IPs, configs and vaults, the second one pointed at the
first's PostgreSQL server, both told about the topology, and the secrets
partitioned the way a shared database and separate clusters require.
This is the test that pins the invariants the individual commits only
enforce locally.

Also documents multi-DC on GCP in the installation guide: the quota
footprint, the install order, the two config/vault/secrets-dir paths, the
topology keys and the DNS scheme, including the per-data-center platform
records the frontend depends on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Signed-off-by: NJona <25478046+NJona@users.noreply.github.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
@NJona NJona self-assigned this Aug 7, 2026
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
@NJona
NJona force-pushed the multi-dc-10-multi-dc-tests-docs branch from 355417a to cc56549 Compare August 7, 2026 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant